An Improved Hierarchical Lossless Text Compression Algorithm

Several improvements to the Bugajski-Russo N-gram algorithm are proposed. When applied to English text these result in an algorithm with comparable complexity and approximately 10 to 30% less rate than the commonly used COMPRESS algorithm. I. The N-Gram Algorithm The N-gram algorithm of Bugajski and Russo [1] is a hierarchical dictionary-type universal lossless source coder for a finite source alphabet. There are several variations, but they share the same basic idea which is described as follows: There is a set of L dictionaries D1, D2, D3, ... , DL, where the level i dictionary, Di, contains words of length 2i-1, D1 contains all possible single characters, and Di ⊂ Di-1 × Di-1, i = 2, ..., L. To encode a source sequence (x1,x2, ...), one finds the largest i such that (x\s( ,1), x\s( ,2), ..., x\s( ,2i-1)) ∈ Di and sends the level identifier i and the index of (x ּ 1 ,x ּ 2 , . . . ,x ּ 2i-1 ) in Di. The level identifier i is encoded with a prefix code and the index is encoded with mi = log |Di| bits, where |Di| denotes the size of the set Di and where all logarithms in this paper have base 2. Thus an N-gram algorithm is a kind of variablelength to variable-length code. The effect of the hierarchical coding is to parse (x1,x2,...) into words whose lengths is a power of two. There are two strategies for developing the dictionaries: one-pass and two-pass. In a one-pass method, the dictionaries are developed while the source sequence is being encoded, as in the LZ78 algorithm [2]. In a two-pass method, a long source sequence is first scanned for the purpose of developing the dictionaries and then the source sequence is encoded as described previously. The dictionaries must themselves be encoded and sent along with the level identifiers and dictionary indices. In this paper, we focus only on two-pass methods. The encoding rate, or simply rate, R of a two-pass method is the summation of the index, dictionary, and level identifier rates; i.e.